Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restrict latest versions to stable highest releases only for all default repositories #2501

Draft
wants to merge 7 commits into
base: master
Choose a base branch
from
Draft

Restrict latest versions to stable highest releases only for all default repositories #2501

wants to merge 7 commits into from

Conversation

walterdeboer
Copy link

@walterdeboer walterdeboer commented Feb 17, 2023 •
edited
Loading

Description

Added a regex to determine unstable releases. Every release in the repository metadata is examined and only stable releases are considered to determine the highest version number, instead of the latest by date.

Before:
image

After:
image

Addressed Issue

closes #2500, fixes #513, fixes #1374

Additional Details

Testing is a bit hard since curernt tests need actual HTTP calls. Extracted a few utility methods for verification.

Cargo and Go report highest stable versions. Updated the other repositories to use the utility methods. All repositories now return stable highest versions when found.

Fixed inaccurate published timestamp. It also got updated when a new lower or unstable version was pushed to the repo. The published timestamp is now set only when the latest version in the metadata equals the latest stable version found so we know for sure the published timestamp belongs to that version.

Centralized all version matching in ComponentVersion, added complex version matching to match semver as well as non-semver and different ecosystems, such as debian and ubuntu, that use epoch numbers in versions, or use labels with ''ubuntu" in it. (#1374)

Checklist

  • I have read and understand the contributing guidelines
  • This PR fixes a defect, and I have provided tests to verify that the fix is effective
  • This PR implements an enhancement, and I have provided tests to verify that it works as intended
  • This PR introduces changes to the database model, and I have added corresponding update logic
  • This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

@sonatype-lift
Copy link
Contributor

sonatype-lift bot commented Feb 18, 2023

🛠 Lift Auto-fix

Some of the Lift findings in this PR can be automatically fixed. You can download and apply these changes in your local project directory of your branch to review the suggestions before committing.1

# Download the patch
curl https://lift.sonatype.com/api/patch/github.com/DependencyTrack/dependency-track/2501.diff -o lift-autofixes.diff

# Apply the patch with git
git apply lift-autofixes.diff

# Review the changes
git diff

Want it all in a single command? Open a terminal in your project's directory and copy and paste the following command:

curl https://lift.sonatype.com/api/patch/github.com/DependencyTrack/dependency-track/2501.diff | git apply

Once you're satisfied, commit and push your changes in your project.

Footnotes

  1. You can preview the patch by opening the patch URL in the browser. ↩

@walterdeboer walterdeboer reopened this Feb 19, 2023
@walterdeboer walterdeboer reopened this Feb 19, 2023
@walterdeboer walterdeboer changed the title Restrict latest versions to stable releases only for maven repositories Restrict latest versions to stable releases only for Composer, Hex, Maven, NPM and NuGet repositories Feb 20, 2023
@walterdeboer walterdeboer changed the title Restrict latest versions to stable releases only for Composer, Hex, Maven, NPM and NuGet repositories Restrict latest versions to stable releases only for Composer, Hex, Maven, NPM, NuGet and PyPi repositories Feb 20, 2023
@walterdeboer walterdeboer changed the title Restrict latest versions to stable releases only for Composer, Hex, Maven, NPM, NuGet and PyPi repositories Restrict latest versions to stable releases only for all repositories Feb 20, 2023
@msymons
Copy link
Member

msymons commented Apr 27, 2023

@walterdeboer, following the release of v4.8.0, this PR now has conflicts that need to be resolved before it can be merged. Please can you fix things? Then the PR can be reviewed.

melba-lopez
melba-lopez approved these changes Jun 27, 2023
View reviewed changes
Copy link
Contributor

@melba-lopez melba-lopez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much @walterdeboer for making these changes. Def helps to make the code more efficient/clean and 💟 the checking of stable versions aspect of it 😸

src/main/java/org/dependencytrack/util/ComponentVersion.java Show resolved Hide resolved
@melba-lopez
Copy link
Contributor

@nscuro there are no merge conflicts and it is updated with the latest changes of base branch.

@walterdeboer walterdeboer force-pushed the master branch 4 times, most recently from 53b816c to 30be928 Compare June 29, 2023 20:42
@melba-lopez melba-lopez added the enhancement New feature or request label Jul 28, 2023
@nscuro nscuro self-assigned this Aug 14, 2023
@melba-lopez
Copy link
Contributor

melba-lopez commented Aug 22, 2023 •
edited
Loading

@walterdeboer can you help resolve the branch conflict?

melba-lopez
melba-lopez approved these changes Aug 22, 2023
View reviewed changes
Copy link
Contributor

@melba-lopez melba-lopez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reviewed extra changes since last approval. Thanks for additional tests @walterdeboer

@nscuro
Copy link
Member

nscuro commented Aug 22, 2023

@melba-lopez @walterdeboer The conflict just resolved itself after merging #2965 :)

@nscuro
Copy link
Member

nscuro commented Oct 16, 2023

Thanks again @walterdeboer for the effort here.

Upon further inspection, there are two major things that prevent us from merging this:

  • I had multiple orgs reach out indicating that they in fact want to see SNAPSHOT / pre-release versions to be considered as latest. This seems to be the case for primarily internal repositories, e.g. where developers should be notified when a newer SNAPSHOT version of an internal library is available. What this means is that at the very least, the feature implemented in this PR should be optional, potentially even configurable on a per-repository basis.
  • This PR touches a very central class, namely ComponentVersion. This class is not only used during repository meta analysis, but also during internal vulnerability analysis. Trying to deal with multiple versioning schemes (Debian, Ubuntu, RPM, etc.) in a single class is highly error-prone and hard to debug. What we're planning to do is to integrate vers (Add support for vers version ranges and ecosystem-specific version comparisons #2826), which includes implementations for ecosystem-specific version matching and comparisons. The plan is to adopt https://github.com/nscuro/versatile for this. Once added, it will be much easier to extend the comparison logic for more versioning schemes.

@nscuro nscuro removed this from the 4.9 milestone Oct 16, 2023
@walterdeboer
Copy link
Author

walterdeboer commented Oct 18, 2023 •
edited
Loading

Thanks for the heads-up @nscuro I'll have a look how to add support for in/excluding pre-releases, i'll get back on that.
ComponentVersion definitely deserves a better foundation. Vers(atile) sounds good!

Walter de Boer and others added 7 commits October 19, 2023 01:39
Restrict latest versions to stable releases only for maven repositories
76eddc9 
Signed-off-by: Walter de Boer <[email protected]>
@nscuro
Review suggestions:
cccef92 
- Compiling regex Patterns to save computational effort
- Documenation

Co-authored-by: Niklas <[email protected]>
Signed-off-by: Walter de Boer <[email protected]>
-develop means unstable
526c42b 
Signed-off-by: Walter de Boer <[email protected]>
Centralized all version matching in ComponentVersion, added complex v…
7a17346 
…ersion matching to match different ecosystems

Signed-off-by: Walter de Boer <[email protected]>
Fix regex ReDOS vulnerability
0ea66d6 
Signed-off-by: Walter de Boer <[email protected]>
Update DB to remove possible unstable versions from analysis cache an…
4e85ef4 
…d repository metadata

Signed-off-by: Walter de Boer <[email protected]>
Extra SemVer tests
6f5ce51 
Signed-off-by: Walter de Boer <[email protected]>
@walterdeboer walterdeboer marked this pull request as draft October 19, 2023 00:43
@undeadly
Copy link

This PR fixes my major complaint with Dependency Track. Please revive this PR and get it merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sonatype-lift sonatype-lift[bot] sonatype-lift[bot] left review comments

@melba-lopez melba-lopez melba-lopez approved these changes

@nscuro nscuro Awaiting requested review from nscuro

Assignees

@nscuro nscuro

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
5 participants
@walterdeboer @msymons @melba-lopez @nscuro @undeadly